Diffusion Approximations for the Constant Step Size Backpropagation Algorithm and Resistance to Local Minima

In this paper we discuss the asymptotic properties of the most commonly used variant of the backpropagation algorithm in which network weights are trained by means of a local gradient descent on examples drawn randomly from a fixed training set, and the learning rate TJ of the gradient updates is held constant (simple backpropagation). Using stochastic approximation results, we show that for TJ ~ 0 this training process approaches a batch training and provide results on the rate of convergence. Further, we show that for small TJ one can approximate simple back propagation by the sum of a batch training process and a Gaussian diffusion which is the unique solution to a linear stochastic differential equation. Using this approximation we indicate the reasons why simple backpropagation is less likely to get stuck in local minima than the batch training process and demonstrate this empirically on a number of examples.